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ABSTRACT 


This paper analyses the size and nature of the errors in GDP forecasts in 
the G7 countries from 1971 to 1995. These GDP short-term forecasts are 
produced by the Organization for Economic Cooperation and Develop- 
ment and by the International Monetary Fund, and published twice a year 
in the Economic Outlook and in the World Economic Outlook, respectively. 
The evaluation of the accuracy of the forecasts is based on the properties 
of the difference between the realization and the forecast. A forecast 
is considered to be accurate if it is unbiased and efficient. A forecast is 
unbiased if its average deviation from the outcome is zero, and it is efficient 
if it reflects all the information that is available at the time the forecast is 
made. Finally, we also examine tests of directional accuracy and offer a 
non-parametric method of assessment. Copyright © 2000 John Wiley & 
Sons, Ltd. 
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INTRODUCTION 


Over the last few years many national and international organizations have produced reports 
aimed at analysing the economic situation and at providing forecasts for the future evolution of 
the main macroeconomic figures. The purpose of this paper is to examine the size and nature of 
the errors in GDP forecasts in the G7 countries from 1971 to 1995. These GDP short-term 
forecasts are produced by the Organization for Economic Cooperation and Development 
(OECD) and by the International Monetary Fund (IMF), and published twice a year in the 
Economic Outlook and in the World Economic Outlook, respectively. 

An analysis of the economic forecasts produced by the OECD, the IMF and by other 
organizations reveals that: (1) the forecasts made by economists linked to different organizations 
and institutions tend to be suspiciously similar; (2) these forecasts are usually incorrect, that is, 
substantial differences arise between the forecasts and the final economic growth; (3) there is a 
generalized belief that the events which occur in the present will be repeated in the future (the 
economists are ‘momentum-followers’ who believe today’s events will reoccur tomorrow); and (4) 


* Correspondence to: Jordi Pons, Department of Econometrics, Statistics and Spanish Economy, University of 
Barcelona, Avda Diagonal, 690, 08034-Barcelona, Spain. E-mail: jpons@eco.ub.es 


CCC 0277—6693/2000/010053—11$17.50 Received December 1997 
Copyright © 2000 John Wiley & Sons, Ltd. Accepted January 1999 


54 J. Pons 


the economic forecasts tend to be very conservative, avoiding taking risks when it comes to 
making diagnoses on the future evolution of an economy. To address these points, this article 
performs an evaluation of the accuracy of the forecasts, and analyses whether these international 
organizations have a tendency to ‘clump’ (make similar forecasts). 

The evaluation of the accuracy of the forecasts is based on the properties of the difference 
between the realization and the forecast. A forecast is considered to be accurate if it is unbiased 
and efficient. A forecast is unbiased if its average deviation from the outcome is zero, and it is 
efficient if it reflects all the information that is available at the time the forecast is made. Finally, 
we also examine tests of directional accuracy and offer a non-parametric method of assessment. 

Several studies of the accuracy of forecasts have appeared in the literature, and the pioneering 
work of Gartaganis and Goldberger (1955), Zellner (1958), Morgenstern (1963) and Stekler 
(1967, 1968) can be mentioned, together with the more recent studies by Howrey (1978, 1984), 
Mankiw et al. (1984), Nordhaus (1985), Mankiw and Shapiro (1986), McNess (1986), Mork 
(1987), Holden and Peel (1988), Artis (1988, 1996), Wallis (1989), Holden (1990, 1997), Stekler 
(1991, 1994), Barrionuevo (1993), Kolb and Stekler (1993), Brodsky and Newbold (1994), Leitch 
and Tanner (1995), Rathjens and Robins (1995) and Graham (1996), all of which refer to the 
United States and United Kingdom economies. 

This paper proceeds as follows. The next section examines GDP short-term forecasts produced 
by the OECD and the IMF and evaluates the accuracy of these forecasts, the third section 
examines tests of directional accuracy and the results are summarized in the final section. 


FORECASTING ACCURACY 


Two sets of projections are examined: the current-year forecasts prepared in the spring of the 
same year, and year-ahead forecasts made in the fall for the following year. The GDP short-term 
forecasts used in this paper are produced by the OECD and the IMF, and published twice a year 
in the Economic Outlook and in the World Economic Outlook, respectively. The sample period is 
1971-95 for the current-year forecasts and 1973-95 for the year-ahead forecasts. 

The OECD forecasts for the current year appear in June, while those of the IMF are published 
in May. On the other hand, the one-year-ahead forecasts are published in December by the 
OECD and in October by the IMF. Hence, it can be said that the forecasts of both organizations 
are done in the same time horizon. In this work are used the OECD forecasts from the Economic 
Outlook of the OECD, and the IMF forecasts are extracted from the World Economic Outlook 
and from Artis (1996). 

Table I presents the values of several traditional measures of goodness of forecast with respect 
to the final value. The statistical indicators used are root mean square error (RMSE), mean 
absolute error (MAE) and Theil’s U.! 


'Theil’s U is calculated according to the following formula: 


VX=(R — FP 


xR? 


where R are the actual values and F the fitted values. 
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Table I. Measures of goodness of forecasts 


Current-year forecast 


OECD IMF 

MAE RMSE U MAE RMSE U 
United States 0.84 1.08 0.29 0.91 1.16 0.32 
Japan 1.23 75 0.35 1.29 1.85 0.37 
Germany 1.10 1.24 0.41 1.28 1.46 0.48 
France 0.81 1.03 0.33 0.87 1.09 0.36 
Italy 1.19 1.59 0.58 1.18 1.51 0.56 
United Kingdom 0.78 0.95 0.37 0.95 1.12 0.41 
Canada 1.08 1.24 0.34 1.18 1.42 0.38 


Year-ahead forecast 


OECD IMF 

MAE RMSE U MAE RMSE U 
United States 0.91 1.08 0.34 1.19 1.54 0.45 
Japan 1.33 2.15 0.47 1.73 2.62 0.52 
Germany 1.55 1.81 0.66 1.54 1.85 0.63 
France 1.13 1.48 0.52 1.19 1.64 0.55 
Italy 1.54 1.95 0.66 1.52 2.02 0.73 
United Kingdom 1.57 1.94 0.79 1.63 2.05 0.84 
Canada 1.52 LPL 0.52 1.49 1.86 0.51 


Note: MAE = mean absolute error, RMSE = root mean square error, U = Theil’s U. 


From the results it can be seen that the forecasts made by the OECD and the IMF show similar 
errors, although in all cases, except for the predictions for the Italian economy in the current 
year, the forecasts by the OECD are slightly better than those of the IMF. Also, the errors 
increase when the forecast of the growth rate is made with a longer time horizon. The summary 
statistics clearly support the propositions that OECD forecasts are superior to the IMF forecasts; 
that current-year forecasts are better than the longer-term year-ahead projections; and the most 
accurate forecasts for the current year are for the United States, France and the United Kingdom, 
and for the year-ahead forecasts for the United States and France. 

Table II provides the values of goodness of forecasts for two samples, where the main sample is 
halved by breaking it at 1983. One of the most important conclusions which can be drawn from 
this research is that it cannot be agreed that in all cases the OECD and IMF forecast accuracy has 
improved in the last few years, because in some countries it can be observed how the forecasts 
made in the first subperiod are higher than those in the second. Finally, the summary statistics 
reviewed here do not afford a basis for a strong verdict either way on whether forecasting errors 
have fallen over the period. These results are consistent with those obtained recently by Artis 
(1996) for the IMF forecasts. 

This paper evaluates the accuracy of a forecast based on the properties of the forecast error. A 
forecast is considered to be accurate if it is unbiased and efficient. A forecast is unbiased if its 
average deviation from the outcome is zero. Bias may be identified with the significance of the 
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Table II. Measures of goodness of forecasts. A comparison of two subperiods 


United States 
Japan 

Germany 

France 

Italy 

United Kingdom 


Canada 


United States 
Japan 

Germany 

France 

Italy 

United Kingdom 


Canada 


1971-1982 
1983-1995 
1971-1982 
1983-1995 
1971-1982 
1983-1995 
1971-1982 
1983-1995 
1971-1982 
1983-1995 
1971-1982 
1983-1995 
1971-1982 
1983-1995 


1973-1982 
1983-1995 
1973-1982 
1983-1995 
1973-1982 
1983-1995 
1973-1982 
1983-1995 
1973-1982 
1983-1995 
1973-1982 
1983-1995 
1973-1982 
1983-1995 


MAE 


1.05 
0.64 
1.75 
0.74 
1.13 
1.07 
0.81 
0.80 
1.75 
0.67 
0.74 
0.81 
0.96 
1.20 


MAE 


0.93 
0.90 
1.78 
1.00 
1.59 
1.51 
1.05 
1.19 
2.42 
0.86 
1.62 
1.53 
1.41 
1.60 


OECD 


RMSE 


1.35 
0.73 
2.37 
0.87 
1.29 
1.19 
1.04 
1.01 
2.12 
0.83 
0.87 
1.02 
1.18 
1.29 


OECD 


RMSE 


1.04 
1.11 
2.92 
1.28 
1.99 
1.66 
1.48 
1.48 
2.70 
1.06 
2715 
1.76 
1.80 
1.75 


U 


0.32 
0.23 
0.38 
0.24 
0.38 
0.46 
0.26 
0.49 
0.68 
0.35 
0.34 
0.40 
0.29 
0.40 


Year-ahead forecast 


U 


0.29 
0.39 
0.51 
0.37 
0.62 
0.71 
0.41 
0.71 
0.76 
0.45 
0.84 
0.74 
0.47 
0.57 


Current-year forecast 


MAE 


0.98 
0.84 
1.93 
0.70 
1.26 
1.31 
0.71 
1.02 
1.64 
0.76 
0.75 
1.14 
1.08 
1.28 


MAE 


1.39 
1.03 
2.26 
1.32 
1.81 
1.33 
0.92 
1.40 
2.15 
1.03 
1.61 
1.65 
1.25 
1.67 


Note: MAE = mean absolute error, RMSE = root mean square error, U = Theil’s U. 


mean forecast error, as indicated by a simple regression of the error on a constant term (Holden 


IMF 


RMSE 


1.33 
0.98 
2.50 
0.93 
1.43 
1.48 
0.91 
1.23 
1.97 
0.90 
0.91 
1.29 
1.42 
1.42 


IMF 


RMSE 


1.82 
1.30 
3.47 
1.68 
2.15 
1.59 
1.45 
1.78 
2.65 
1.33 
2.17 
1.95 
1.88 
1.84 


and Peel, 1990). The test for biasedness is based on the regression expressed as: 


where e, is the forecast error. In addition, unbiasedness is also important because it is a necessary 


condition for efficiency. 
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U 


0.31 
0.33 
0.40 
0.29 
0.40 
0.61 
0.23 
0.63 
0.64 
0.39 
0.31 
0.52 
0.33 
0.46 
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Table HI. Test for biasedness 


Current-year forecast Year-ahead forecast 

OECD IMF OECD IMF 

United States 0.01 —0.02 —0.09 0.14 
(0.97) (0.93) (0.71) (0.67) 

Japan 0.51 0.62 0.73 1.17 
(0.15) (0.11) (0.11) (0.03) 

Germany —0.16 —0.06 0.04 0.43 
(0.52) (0.84) (0.92) (0.27) 

France 0.06 0.07 0.11 —0.31 
(0.78) (0.75) (0.72) (0.08) 

Italy —0.51 —0.38 —0.43 —0.22 
(0.11) (0.21) (0.30) (0.61) 

United Kingdom —0.47 —0.34 —0.22 0.13 
(0.01) (0.13) (0.61) (0.78) 

Canada —0.24 —0.22 —0.03 0.24 
(0.34) (0.44) (0.94) (0.54) 


Notes: A test for biasness is based on the regression e, = y + u,. 
The figures in parentheses are the significance level of the f- statistic for y = 0. 


In Table III the value of the mean forecast error, y, is shown and in parentheses are given the 
significance levels at which the null might be rejected. Generally, the evidence is that these 
forecasts are not biased, except in the case of the United Kingdom (OECD forecasts in the 
current year) and those for Japan made by the IMF in the fall for the following year. Also, a 
constant pattern of overestimation/underestimation (suggesting that there may be a widespread 
error of output growth optimism) is not detected. 

When the forecast error is defined as the difference between the forecast and the real value, a 
positive (negative) value of y and significantly different from zero points to an overestimation 
(underestimation) of the GDP growth. Hence, in the Japanese case is seen for the IMF a trend to 
overvalue the GDP growth in the forecasts made one year ahead, and in the United Kingdom’s 
case there is confirmed an undervaluation of its growth for OECD forecasts made in the current 
year. 

Forecast efficiency implies that the deviation between the outcome and the projection is not 
related to information available at the time that the projections were made (Holden and Peel, 
1990; Barrionuevo, 1996). This condition is tested by measuring the statistical significance of the 
co-movements between the deviation of the outcome of the forecast and the forecast itself 
(B-test), and the co-movement between the deviation of the outcome of the forecast in the current 
period and that in the previous period (p-test). 6 is estimated by a least-squares regression of the 
forecast error on a constant and the forecast (equation (2)), and p is estimated by a regression of 
the current-period forecast error on a constant and the previous-period error (equation (3)). 
Thus, a condition for efficiency is that both f and p be zero. 


e,= a +f8-F, +4, (2) 
Cp = % TP ey) TU, (3) 


where e, and F, denote forecast error and forecast, respectively. 
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These equations provide a useful distinction between different forms of inefficiency (Barrio- 
nuevo, 1996). If fis zero and p is different from zero the forecast is inefficient because the errors of 
the past are repeated in the present, and hence forecasts could be improved by adjusting them by p. 
If 6 is different from zero and p is zero, the inefficiency arises because the model used to derive the 
forecast is not the minimum variance model. Finally, if 6 and p are both not equal to zero, the 
inefficiency is partly due to the way in which new information is incorporated into projected values 
and partly because the errors of the past are repeated in the present. 

In Table IV we analyse the efficiency of the forecasts made by the OECD and the IMF in both 
temporal horizons considered. If a significance level « = 0.05 is established, only in the case of 
forecasts made by the IMF one year ahead for Japan can it be rejected that 6 = 0 and p = 0, that 
is, the forecasts for the Japanese economy are inefficient, due to the fact that the past errors are 
repeated in the present (p 4 0) and do not incorporate all available information at the time of 
making the prediction (f 4 0). The forecast made by the IMF in the current year for France can 
also be considered as inefficient due to p 4 0, that is, the past errors are repeated in the present 
forecasts. 

A third case in which forecasts can also be considered as inefficient is when 6 4 0 and p = 0. 
This means that forecasts do not include all available information when they are made. In this 
situation are the OECD forecasts in the current year for the United States and the United 
Kingdom; the OECD forecasts in the current year and one year ahead for Japan; and the IMF 
forecasts in the current year for Japan. 

It can be said that forecasts made by the OECD and the IMF for both temporal horizons are 
efficient, except in the case of Japan. In the same way, the OECD forecasts in the current year for 
the United States and the United Kingdom do not embody all available information when they 
are computed, and those for France by the OECD in the current year repeat the errors committed 
in the past. Among the cases studied only in two can it be accepted that p = 0 (the errors in the 
past are repeated in the present) and in six that 6 = 0 (forecasts do not embody all available 
information at the moment of being made). Curiously, they are slightly higher when efficiency is 
studied by forecasts made one year ahead, while it is not possible to differentiate between 
forecasts by the OECD and the IMF. 


DIRECTIONAL ACCURACY 


In the previous section the sizes of forecast errors have only been considered without considering 
their direction. An error in the forecast sign direction consists of an acceleration (deceleration) of 
growth forecast instead of the real deceleration (acceleration). The directional accuracy data can 
be arranged in a 2 x 2 contingency table. In Table V the columns are formed for forecasts of 
positive and negative changes and the rows for positive and negative realizations. It is obvious 
that the objective is that the sum of entries in the two cells of the leading diagonal (7%, + 1,,) 
should be high. 
A simple 7” test of independence can be supplied in this framework: 
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Table IV. Tests for efficiency 


B-test 

Current-year forecast Year-ahead forecast 

OECD IMF OECD IMF 

United States 0.12 0.11 —0.01 0.11 
(0.04) (0.09) (0.93) (0.28) 

Japan 0.21 0.21 0.24 0.32 
(0.00) (0.00) (0.01) (0.00) 
Germany 0.05 0.08 0.09 0.15 
(0.64) (0.40) (0.51) (0.28) 
France 0.11 0.09 0.16 —0.08 
(0.09) (0.21) (0.15) (0.19) 
Italy —0.11 —0.16 —0.00 —0.04 
(0.33) (0.16) (0.99) (0.78) 
United Kingdom —0.18 —0.08 —0.02 0.01 
(0.01) (0.35) (0.90) (0.94) 
Canada —0.07 —0.04 —0.05 0.03 
(0.28) (0.60) (0.65) (0.82) 
p-test 

Current-year forecast Year-ahead forecast 

OECD IMF OECD IMF 

United States —0.02 —0.02 0.07 —0.21 
(0.94) (0.93) (0.34) (0.34) 

Japan —0.22 —0.02 0.24 0.42 
(0.24) (0.90) (0.26) (0.04) 

Germany —0.08 0.24 —0.06 0.16 
(0.71) (0.25) (0.78) (0.48) 

France 0.36 0.42 0.14 0.07 
(0.07) (0.04) (0.51) (0.34) 

Italy 0.29 0.20 0.26 0.22 
(0.16) (0.31) (0.23) (0.29) 

United Kingdom 0.35 0.26 0.22 0.35 
(0.08) (0.22) (0.30) (0.09) 

Canada 0.25 0.06 0.10 0.05 
(0.24) (0.80) (0.66) (0.81) 


Notes: The f-test is based on the regression e, = «, + B- F,+u, and the p-test on the regression 
€, =%+p-e,_,+u,. The figures in parentheses are the significance levels of the r-statistic for 
B=0orp=0. 


where the null is that forecasts and realizations are independent (see Schnader and Stekler, 1990; 
Stekler, 1994; Leitch and Tanner, 1995; Artis, 1996). Rejection of this hypothesis implies that 
there is a significant association between the signs of realizations and forecasts. This analysis has 
been based exclusively on the signs of the predicted and actual changes and has, consequently, 
ignored quantitative errors (small and large errors are treated equally in this classification 
scheme). 
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Table V. Contingency table form for directional accuracy 


Forecast 
AF>0 AF <0 Subtotal 
Realization 
AR>0O N09 No} No, 
AR <0 N19 ny ny 
Subtotal No ny N 


Table VI presents the number of correct and incorrect direction forecasts that were made by 
the two organizations. The accuracy of the forecasts generally increases with a decline in the 
forecast horizon. Both organizations had similar forecasting experiences over the time period 
which was considered, although it can be said that those made by the OECD are slightly better. 
In the same way the independence between the forecasts’ sign and the realizations can be accepted 
in the case of Italy in the OECD forecasts made one year ahead and in the IMF forecasts in both 
temporal horizons considered, especially the IMF forecasts for the Japanese economy in both 
temporal horizons. 


CONCLUSIONS 


This paper evalutes the accuracy of forecasts in the G7 countries produced by the OECD and by 
the IMF. Two sets of projections are examined: the current-year forecasts prepared in the spring 
of the same year, and year-ahead forecasts made in the fall for the following year. The results of a 
number of empirical tests support the following conclusions: 


e The summary statistics clearly support the propositions that OECD forecasts are superior to 
the IMF forecasts, and that current-year forecasts are better than the longer-term year-ahead 
projections. 

e The results reviewed in Table H do not afford a basis for a strong verdict either way on whether 
forecasting errors have fallen over the period. 

e A constant pattern of overestimation/underestimation (suggesting that there may be a 
widespread error of output growth optimism) is not detected. 

e OECD and IMF forecasts in both temporal horizons can be considered as efficient, except in 
the case of Japan. In the same way, the OECD forecasts for the current year for the United 
States and the United Kingdom do not embody all available information at the time of being 
made, and those for France by the OECD in the current year repeat the errors of the past. 

e When forecast efficiency is studied it is concluded that forecasts made one year ahead are better 
than those of the current year, while it is not possible to differentiate between OECD and IMF 
proposals. 

e Analysis based exclusively on the signs of predicted and actual changes shows that the 
independence hypothesis can be accepted between the forecasts and the actual values in the 
Italian case (OECD forecasts one year ahead and IMF in both temporary horizons) and Japan 
(the IMF in both temporal horizons), while in other countries this hypothesis can be rejected. 
Moreover, GDP growth direction forecasts made by the OECD are slightly higher than those of 
the IMF, and, finally, accuracy of the forecasts generally increases with a decline in the forecast 
horizon. 
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Table VI. Contingency table of directional forecast accuracy 
Current-year forecast—OECD 


AR>0,AF>0 AR>0,AF<0 AR<0,AF>0 AR<0,AF<0 ¢ 


United States 8 2 1 13 14.31 
Japan 13 1 2 8 14.314 
Germany 9 2 2 11 11.40? 
France 8 2 2 12 11.114 
Italy 9 3 3 9 6.748 
United Kingdom 12 3 0 9 15.38? 
Canada 10 1 0 13 21.288 


Current-year forecast IMF 


AR>0,AF>0 AR>0,AF<0 AR<0,AF>0 AR<0,AF<0 ¢C 


United States 8 2 2 12 11.408 
Japan 11 3 3 7 6.58° 
Germany 9 2 3 10 9.00° 
France 7 3 2 12 8.36? 
Italy 7 5 4 8 1.92 

United Kingdom 10 5 0 9 1s 
Canada 9 2 1 12 14.49 


Year-ahead forecast OECD 


AR>0,AF>0 AR>0,AF<0 AR<0,AF>0 AR<0,AF<0 Va 


United States 6 2 0 14 15.654 
Japan 9 4 3 6 1.96 

Germany 7 2 3 10 7.338 
France 7 2 4 9 5.49» 
Italy 6 4 6 6 0.43 

United Kingdom 9 4 1 8 7.998 
Canada 8 2 1 11 12.61? 


Year-ahead forecast—IMF 


AR>0,AF>0 AR>0,AF<0 AR<0,AF>0 AR<0,AF<0 C 


United States 5 3 3 11 4.71% 
Japan 9 4 4 5 0.88 

Germany 8 1 4 9 8.07" 
France 7 2 5 8 3.97 
Italy 6 4 4 8 2.11 

United Kingdom 9 4 D, 7 5.32 
Canada 8 2 1 11 12.614 


Notes: F and R denote forecast and realization, respectively. AR = R,— R,_, and AF = F, — R,_,. (a) indicates that the 
null hypothesis of independence can be rejected at the 1% significance level and (b) at the 5% significance level. 
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